Neural cortex

ABSTRACT

A neural network system includes a random access memory (RAM); and an index-based weightless neural network with a columnar topography; wherein patterns of binary connections and values of output nodes&#39; activities are stored in the RAM. Information is processed by pattern recognition using the neural network by storing a plurality of output patterns to be recognised in a pattern index; accepting an input pattern and dividing the input pattern into a plurality of components; and processing each component according to the pattern index to identify a recognised output pattern corresponding to the input pattern.

FIELD OF INVENTION

[0001] The invention relates to an index-based neural network and to amethod of processing information by pattern recognition using a neuralnetwork. It relates particularly but not exclusively to a neural networkcomputer system which has an index-based weightless neural network witha columnar topography, and to a method whereby an input pattern isdivided into a plurality of components and each component is processedaccording to a single pattern index to identify a recognised outputpattern corresponding to the input pattern.

BACKGROUND TO THE INVENTION

[0002] An artificial neural network is a structure composed of a numberof interconnected units typically referred to as artificial neurons.Each unit has an input/output characteristic and implements a localcomputation or function. The output of any unit can be determined by itsinput/output characteristic and its interconnection with other units.Typically the unit input/output characteristics are relatively simple.

[0003] There are three major problems associated with artificial neuralnetworks, namely: (a) scaling and hardware size practical limits; (b)network topology; and (c) training. The scaling and hardware sizeproblem arises because there is a relationship between applicationcomplexity and artificial neural network size, such that scaling toaccommodate a high resolution image may require hardware resources whichexceed practical limits.

[0004] The network topology problem arises due to the fact that,although the overall function or functionality achieved is determined bythe network topology, there are no clear rules or design guidelines forarbitrary application.

[0005] The training problem arises because training is difficult toaccomplish.

[0006] The n-Tuple Classifier has been proposed in an attempt to addressthese problems. This classifier was the first suggested RAM-based neuralnetwork concept. The first hardware implementation of the n-TupleConcept was the WISARD system developed at Brunel University around 1979(see “Computer Vision Systems for Industry: Comparisons”, appearing asChapter 10 in “Computer Vision Systems for Industry”, I Alexander, TStonham and B Wilkie, 1982). The WISARD system belongs to the class ofRAM-based weightless neural networks. This style of neural networkaddresses the problem of massive computations-based training by writingthe data into a RAM-network and the problem of topology by suggesting auniversal RAM-based network structure. However, the network topology ofWISARD-type universal structure does not simulate the higher levels ofneuronal organization found in biological neural networks. This leads toinefficient use of memory with the consequence that the problem ofscaling still remains acute within RAM-based neural networks, and theapplication range of the WISARD-technology is limited.

[0007] Another example of neural networks that overcomes the problem oftraining by a simple memorization task is Sparse Distributed Memory (PKanerva, 1998, “Sparse Distributed Memory”, Cambridge, Mass.: NITPress). However, a problem with the Sparse Distributed Memory, as withthe WISARD system, is a large memory size. Another disadvantage of theSparse Distributed Memory is its computational complexity. This isbecause for this type of memory, an input word must be compared to allmemory locations.

[0008] N-Tuple classification systems use a method of recognitionwhereby an input to the neural network is divided into a number ofcomponents (n-Tuples) with each component compared to a series ofcomponent look-up tables. Normally there is an individual look-up tablefor each component. The network then processes each component in lightof a large number of look-up tables to determine whether there has beena match. Where a match occurs for a component then that indicates thatthe component has been recognised. Recognition of each of the componentsof an input leads to recognition of the input.

[0009] The presence of a number of look-up tables results in apotentially large memory size. The memory size required is proportionalto the number of components which the network may identify. This canresult in a substantial increase in memory where the pattern sizeincreases. For example, an artificial neural network might be designedfor an image processing application, initially using an n×n image, wheren=128. This is a relatively low-resolution image by today's standards.Where the image to be processed increases from n=128 to n=2048 thenumber of neurons, the size of the network, increases by a factor of256. This increase in memory results in the requirement for networkexpansion potentially requiring additional hardware modular blocks.Where the resolution of the image increases a point is quickly reachedwhere the scaling to accommodate a high resolution image is beyond apractically achievable memory limit.

[0010] An object of the present invention is to address, overcome oralleviate some or all of the disadvantages present in the prior art.

SUMMARY OF THE INVENTION

[0011] According to a first aspect of the invention, there is provided aneural network system including:

[0012] (a) a random access memory (RAM); and

[0013] (b) an index-based weightless neural network with a columnartopography; wherein patterns of binary connections and values of outputnodes' activities are stored in the RAM.

[0014] Preferably, the neural network system comprises a computerhardware component.

[0015] In a preferred form the neural network system has potential forscaling. Scaling may be achieved in any suitable manner. It is preferredthat systematic expansion is achieved by increasing the size of the RAM.

[0016] The neural network may be trained in any suitable manner. It ispreferred that the neural network is trained by writing of data into theRAM, and network topology emerges during the training.

[0017] It is preferred that performance of the neural network isadjustable by changing decomposition style of input data, and therebychanging dynamic range of input components.

[0018] It is preferred that input components to the neural networkaddress a single common index.

[0019] According to a second aspect of the invention, there is provideda method of processing information by pattern recognition using a neuralnetwork including the steps of—

[0020] (a) storing a plurality of output patterns to be recognised in apattern index;

[0021] (b) accepting an input pattern and dividing the input patterninto a plurality of components;

[0022] (c) processing each component according to the pattern index toidentify a recognised output pattern corresponding to the input pattern.

[0023] Preferably each output pattern is divided into a plurality ofrecognised components with each recognised component being stored in thepattern index for recognition. The index preferably consists of columnswith each column corresponding to one or more recognised components.Preferably the index is divided into a number of columns which is equalto or less than the number of recognised components. More preferably,the index is divided into a number of columns which is equal to thenumber of recognised components.

[0024] The method may further include the steps of each input componentbeing compared to the corresponding recognised component column, and ascore being allocated to one or more recognised components. Preferablythe score for each recognised component of a pattern is added and therecognised pattern with the highest score is identified as the outputpattern.

BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The invention will now be described in further detail byreference to the attached drawings which show example forms of theinvention. It is to be understood that the specificity of the followingdescription does not limit the generality of the foregoing disclosure.

[0026]FIG. 1 is an index table illustrating processing of an inputaccording to one embodiment of the invention.

[0027]FIG. 2 is a schematic block diagram illustrating processing of aninput according to an embodiment of the invention.

[0028]FIG. 3 is a schematic block diagram illustrating processing of anoutput according to an embodiment of the invention.

DETAILED DESCRIPTION

[0029] The invention can be implemented through the use of a neural cardbuilt with the use of standard digital chips. The invention is anindex-based weightless neural network with a columnar topology thatstores in RAM the patterns of binary connections and the values of theactivities of output nodes. The network offers:

[0030] (a) Scaling potential: Systematic expansion of the neural networkcan be achieved not by adding extra modular building blocks as inprevious artificial neural networks, but by increasing the RAM size toinclude additional columns or by increasing the height of the index. Forexample, 16 million connections can be implemented with a 64 MB RAM.

[0031] (b) The required memory size is reduced by a factor of N, whencompared with previous n-Tuple systems such as the WISARD system, with Nbeing the number of input components (n-Tuples). This is because then-Tuple Classifier requires N look-up tables, whereas the presentinvention requires only one common storage.

[0032] (c) The network topology emerges automatically during thetraining.

[0033] (d) Training is reduced to writing of data into RAM.

[0034] (e) The performance can easily be adjusted by changing thedynamic range of input components, which can be achieved by changing thedecomposition style of input data.

[0035] A device made according to the present invention is hereinafterreferred to as a Neural Cortex. Both traditional artificial neuralnetworks and traditional RAM-based artificial neural networks arenetworks of neuron-like computing units. However, the computing units ofthe human brain are multi-neuron cortical columns. A general view of thesingle common index on which the present invention is based can best bedescribed as a collection of vertical columns, wherein the signalspropagate in a bottom-to-top fashion.

[0036] Unlike traditional RAM-based neural networks, the Neural Cortexoperates not by memorizing the names of classes in component look-uptables but by creating and memorizing an index (a linked datarepresentation) of input components. This index contains the names ofclasses (class reference numbers) and is created on training.

[0037] On retrieval, the Neural Cortex, like the n-Tuple Classifier,sums up the names activated by input components. The summing operationprovides the generalizing ability typical of neural networks. However,unlike the n-Tuple Classifier, where a “winner-takes-all” strategy isemployed, the Neural Cortex employs a “winners-take-all” strategy. Thisis not a matter of preference but a necessity brought about by using asingle common storage. In case of the n-Tuple Classifier, each inputcomponent (n-tuple) addresses its own look-up table. In case of theNeural Cortex, all input components address a single common index. Thisbrings about a dramatic decrease in memory size. The absence of a singlecommon index in both the n-Tuple Classifier and the Sparse DistributedMemory systems explains why previous RAM-based neural networks haddifficulties in terms of memory requirements whose large sizesignificantly limited the application range.

[0038] Further, a single common index is an efficient solution to theneural network expansion problem. As has been indicated above, bothtraditional artificial neural networks and traditional RAM-basedartificial neural networks have scaling difficulties when theapplication size grows. For instance, if the image size grows from128×128 pixels to 2048×2048 than a traditional artificial neuralnetworks will need a 256-fold increase in memory because the number ofn-tuples increases by the factor of 256=2048*2048/128*128. Howeverparadoxically in the same situation, the Neural Cortex size according tothe present invention may remain unchanged because still only one commonindex is used.

[0039] The present invention creates a single pattern index of inputcomponents. The index contains the output components and is created bystoring the output pattern and training the neurons to recognise thepattern stored within the pattern index.

[0040] An output pattern S is decomposed into N number of components S₁,S₂, S_(N) such that each component S₁ is interpreted as the address of acolumn from the index. Each column stores the reference number of thosepatterns which 20 have the value S_(i) in one or more of theircomponents; each column does not contain more than one sample of eachreference number. When an input I is received this is divided into anumber of components I₁, I₂, . . . , I_(x). Each input component I₁ toI_(x) is processed by the network by comparing the component with thepattern index. Where a component of the input I₁ matches a component ofthe output S_(i) then each reference number listed in the column ofS_(i) has a score of one added to its total. This process is repeatedfor each of the input components. The scores are then added to determinethe winner. The winner is the reference number with the greatest score.The reference number, corresponding to a recognised output pattern, isrecognised by the network.

[0041] An example of the pattern index is illustrated in FIG. 1. Thisfigure illustrates where the index has been trained or programmed torecognise three words “background”, “variable” and “mouse”. In thisfigure the words are assigned the reference numbers 1, 2 and 3respectively. The output patterns are letters from “a” to “z” with theseincluded as columns within the index. When an input is received by thenetwork each of the components of the input is processed by reference tothis single pattern index. In this example the input is in the form ofthe word “mouse”. This input is subsequently broken down into fiveletters. Each letter is processed in the network by using the index. Thesimultaneous nature of processing undertaken in the network can ensurethat processing of each component is undertaken virtuallysimultaneously. The following processing is undertaken—

[0042] (a) the component of the input “m” is processed and in this caseone point is added to the score attributable to variable 3;

[0043] (b) the component input “o” is processed and one point is addedto variable 1 and 3;

[0044] (c) the component input “u” is processed and one point isattributable to variable 1 and 3;

[0045] (d) the component input “s” is processed and one point isattributable to variable 3;

[0046] (e) the component input “e” is processed and one point isattributable to variable 2 and 3.

[0047] The network then sums up the points attributable to eachvariable. In this instance variable 1 has a score of 2, variable 2 ascore of 1 and variable 3 a score of 5. The variable with the highestscore is determined to be the winner and hence identified. The variable3 which has a score of 5, corresponding to the word “mouse”, istherefore considered to be identified.

[0048] In case of standard RAM, two different address words always pointtowards two different memory locations. This is no longer true in caseof the Neural Cortex. For example, if the input pattern has threecomponents (a, b, c) and the component dynamic range is 1 byte then thepatterns (a,c,b), (b,a,c), (b,c,a), (c,a,b), (c,b,a) will produce thesame score equal to 3 because the Neural Cortex is invariant withrespect to permutations. The invariance is caused by the fact that allcomponents (n-tuples) address a single common storage. The commonstorage collapses the N-dimensional space into a one-dimensional spacethus creating the permutational invariance, which is the price to bepaid for dramatic reduction in memory size as compared to traditionalRAM-based neural networks. This invariance is the key to the NeuralCortex. At the same time, it is the beauty of the approach because thisinvariance becomes practically harmless when the component dynamic rangeis increased. For the above example, by using the 2 bytes dynamic range,where the pattern (a,b,c) is converted into the 2 component pattern (ab,bc), the following scores will be obtained: (a,b,c)=>2, (a,c,b)=>0,(b,a,c)=>0, (b,c,a)=>1, (c,a,b)=>1, (c,b,a)=>0, so that the pattern(a,b,c) will be identified correctly.

[0049] In general case, the conversion of the n-component input pattern(s₁, s₂, s_(N)) into a new pattern (c₁, c₂, . . . , c_(M)) whosecomponents have a greater dynamic range and M<N is preferably done bythe software driver of the Neural Cortex card.

[0050] This conversion can be referred to as the C(haos)-transform, ifit converts the sequence of all input patterns into a one-dimensionalchaotic iterated map. The sufficient condition for the absence ofidentification ambiguity is that the sequence of all C-transformed inputpatterns is a chaotic iterated map. This is true because in this caseall pattern components will be different, which leaves no room foridentification ambiguity. In fact, this condition is too strong becauseit is sufficient that any two patterns differ in one component, atleast. For practical purposes a good approximation of the C-transformcan be achieved by increasing components' dynamic range to 2 bytes, 3bytes, etc. when 2, 3 or more components are joined together, e.g.,(a,b,c) is converted into the 2 component pattern (ab, be).

[0051] A Neural Cortex read-cycle block-diagram is shown in FIG. 2. Theblocks ‘Roots’, ‘Links’, ‘Names’, ‘Score’ are RAM-devices. Σ is asummer. T-logic is a terminating logical device.

[0052] 1. Each pattern component (A-Word) is passed to the address busof the ‘Roots’ RAM.

[0053] 2. The output value R of the ‘Roots’ RAM is passed to the addressbus of the ‘Links’ RAM.

[0054] 3. The output value L of the ‘Links’ RAM is passed to the addressbus of the ‘Names’ RAM.

[0055] 4. And, finally, the output value N of the ‘Names’ RAM is passedto the address bus of the ‘Score’ RAM.

[0056] If L is 0 then the T-logic terminates the process. Otherwise, the‘Score’ RAM content found at address N that is determined by the outputof the ‘Name’ RAM is incremented by the value of 1. Next, the ‘Links’RAM output is fed back to the ‘Links’ RAM address bus. The processrepeats itself from point 3.

[0057] A Neural Cortex write-cycle block-diagram is shown in FIG. 3. Theblocks ‘Roots’, ‘Links’, ‘Names’, are RAM-devices. CU is the controlunit.

[0058] 1. Each pattern component A is passed to the address bus of the‘Roots’ RAM.

[0059] 2. The output value R of the ‘Roots’ RAM is passed to theaddress-bus of the ‘Links’ RAM.

[0060] 3. The output value L of the ‘Links’ RAM is passed to theaddress-bus of the ‘Names’ RAM. The output value of the ‘Names’ RAM isdenoted by N, and the current pattern name by P.

[0061] 4. The values R, L, N and P are passed to the control unit, whichutilizes the following logic. If L is 0 then the control unit makes adecision (point 5) on updating ‘Roots’, ‘Links’ and ‘Names’ RAM.Otherwise, L is fed back to the ‘Links’ RAM address bus. The processrepeats itself from point 3.

[0062] 5. Decision Logic:

[0063] a) if N=P, terminate the process;

[0064] if R=0, increment the counter value C by 1,

[0065] write C to ‘Roots’ RAM at address A,

[0066] write C to ‘Links’ RAM at address R,

[0067] write P to ‘Names’ RAM at address L,

[0068] if R>0 & L=0, increment the counter value C by 1,

[0069] write C to ‘Links’ RAM at address R,

[0070] write P to ‘Names’ RAM at address L,

[0071] b) terminate the process.

[0072] Performance of the Neural Cortex can be adjusted in terms ofmemory size and read/write times. Normally, storage and recall timesincrease when the number of classes grows as the training continues.Additional classes increase the amount of reference numbers that arestored in index columns and, therefore, the amount of index cells thathave to be accessed. As a remedy, one can increase the dynamic range Dof input pattern components. This increases the number of index columnsbecause the index address space is equal to D. As a result, the sameamount of reference numbers will be spread upon the greater area, which,in turn, decreases the average index height H.

[0073] The processing time on storage and recall is proportional to thenumber of accessed memory cells, which is proportional to HN. Here, N isthe number of the pattern components. As D increases, the processingtime approaches O(N). This follows from the fact that H is inverseproportional to D.

[0074] The memory size is proportional to HD. However, H grows/decreasesfaster than D. Hence, adjusting the dynamic range D can efficientlycontrol the memory size. In the worst case, the Neural Cortex size doesnot exceed CD, where C is the number of pattern classes, which isbecause the Neural Cortex has only one “look-up-table”. On the otherhand, the memory size of a traditional RAM-based artificial neuralnetwork is CDN because for this type of artificial neural network thenumber of input look-up-tables is equal to the number N of input patterncomponents.

[0075] It is to be understood that various modifications, alterationsand/or additions may be made to the parts previously described withoutdeparting from the ambit of the present invention.

1. A neural network system including: (a) a random access memory (RAM);and (b) an index-based weightless neural network with a columnartopography; wherein patterns of binary connections and values of outputnodes' activities are stored in the RAM.
 2. A neural network systemaccording to claim 1 wherein the system comprises a computer hardwarecomponent.
 3. A neural network system according to claim 1 or claim 2wherein systematic expansion is achieved by increasing the size of theRAM.
 4. A neural network system according to any one of claims 1 to 3wherein the neural network is trained by writing of data into the RAM,and network topology emerges during the training.
 5. A neural networksystem according to any one of claims 1 to 4 wherein performance isadjustable by changing decomposition style of input data, and therebychanging dynamic range of input components.
 6. A neural network systemaccording to any one of claims 1 to 5 wherein input components address asingle common index.
 7. A method of processing information by patternrecognition using a neural network including the steps of— (a) storing aplurality of output patterns to be recognised in a pattern index; (b)accepting an input pattern and dividing the input pattern into aplurality of components; (c) processing each component according to thepattern index to identify a recognised output pattern corresponding tothe input pattern.
 8. A method according to claim 7 wherein each outputpattern is divided into a plurality of recognised components with eachrecognised component being stored in the pattern index for recognition.9. A method according to claim 8 wherein the index consists of columnswith each column corresponding to one or more recognised components. 10.A method according to claim 9 wherein the index is divided into a numberof columns which is equal to or less than the number of recognisedcomponents.
 11. A method according to claim 9 wherein the index isdivided into a number of columns which is equal to the number ofrecognised components.
 12. A method according to any one of claims 8 to10 wherein each input component is compared to the correspondingrecognised component column and a score is allocated to one or morerecognised components.
 13. A method according to claim 12 wherein thescore for each recognised component of a pattern is added and therecognised pattern with the highest score is identified as the outputpattern.